90 research outputs found
Importance Sampling: Intrinsic Dimension and Computational Cost
The basic idea of importance sampling is to use independent samples from a
proposal measure in order to approximate expectations with respect to a target
measure. It is key to understand how many samples are required in order to
guarantee accurate approximations. Intuitively, some notion of distance between
the target and the proposal should determine the computational cost of the
method. A major challenge is to quantify this distance in terms of parameters
or statistics that are pertinent for the practitioner. The subject has
attracted substantial interest from within a variety of communities. The
objective of this paper is to overview and unify the resulting literature by
creating an overarching framework. A general theory is presented, with a focus
on the use of importance sampling in Bayesian inverse problems and filtering.Comment: Statistical Scienc
Markov chain Monte Carlo for exact inference for diffusions
We develop exact Markov chain Monte Carlo methods for discretely-sampled,
directly and indirectly observed diffusions. The qualification "exact" refers
to the fact that the invariant and limiting distribution of the Markov chains
is the posterior distribution of the parameters free of any discretisation
error. The class of processes to which our methods directly apply are those
which can be simulated using the most general to date exact simulation
algorithm. The article introduces various methods to boost the performance of
the basic scheme, including reparametrisations and auxiliary Poisson sampling.
We contrast both theoretically and empirically how this new approach compares
to irreducible high frequency imputation, which is the state-of-the-art
alternative for the class of processes we consider, and we uncover intriguing
connections. All methods discussed in the article are tested on typical
examples.Comment: 23 pages, 6 Figures, 3 Table
Scalable inference for crossed random effects models
We develop methodology and complexity theory for Markov chain Monte Carlo algorithms used in inference for crossed random effect models in modern analysis of variance. We consider 15 a plain Gibbs sampler and a simple modification we propose here, a collapsed Gibbs sampler. Under some balancedness assumptions on the data designs and assuming that precision hyperparameters are known, we demonstrate that the plain Gibbs sampler is not scalable, in the sense that its complexity is worse than proportional to the number of parameters and data, but that the collapsed Gibbs sampler is scalable. In simulated and real datasets we show that the explicit 20 convergence rates our theory predicts match remarkably the computable but non-explicit rates in cases where the design assumptions are violated. We also show empirically that the collapsed Gibbs sampler, extended to sample precision hyperparameters, outperforms significantly, often by orders of magnitude, alternative state of the art algorithms. Supplementary material includes some proofs, additional simulations, implementation details and the R code to implement the 25 algorithms considered in the article
Variational approximation for mixtures of linear mixed models
Mixtures of linear mixed models (MLMMs) are useful for clustering grouped
data and can be estimated by likelihood maximization through the EM algorithm.
The conventional approach to determining a suitable number of components is to
compare different mixture models using penalized log-likelihood criteria such
as BIC.We propose fitting MLMMs with variational methods which can perform
parameter estimation and model selection simultaneously. A variational
approximation is described where the variational lower bound and parameter
updates are in closed form, allowing fast evaluation. A new variational greedy
algorithm is developed for model selection and learning of the mixture
components. This approach allows an automatic initialization of the algorithm
and returns a plausible number of mixture components automatically. In cases of
weak identifiability of certain model parameters, we use hierarchical centering
to reparametrize the model and show empirically that there is a gain in
efficiency by variational algorithms similar to that in MCMC algorithms.
Related to this, we prove that the approximate rate of convergence of
variational algorithms by Gaussian approximation is equal to that of the
corresponding Gibbs sampler which suggests that reparametrizations can lead to
improved convergence in variational algorithms as well.Comment: 36 pages, 5 figures, 2 tables, submitted to JCG
Posterior-based proposals for speeding up Markov chain Monte Carlo
Markov chain Monte Carlo (MCMC) is widely used for Bayesian inference in
models of complex systems. Performance, however, is often unsatisfactory in
models with many latent variables due to so-called poor mixing, necessitating
development of application specific implementations. This paper introduces
"posterior-based proposals" (PBPs), a new type of MCMC update applicable to a
huge class of statistical models (whose conditional dependence structures are
represented by directed acyclic graphs). PBPs generates large joint updates in
parameter and latent variable space, whilst retaining good acceptance rates
(typically 33%). Evaluation against other approaches (from standard Gibbs /
random walk updates to state-of-the-art Hamiltonian and particle MCMC methods)
was carried out for widely varying model types: an individual-based model for
disease diagnostic test data, a financial stochastic volatility model, a mixed
model used in statistical genetics and a population model used in ecology.
Whilst different methods worked better or worse in different scenarios, PBPs
were found to be either near to the fastest or significantly faster than the
next best approach (by up to a factor of 10). PBPs therefore represent an
additional general purpose technique that can be usefully applied in a wide
variety of contexts.Comment: 54 pages, 11 figures, 2 table
Non-parametric Bayesian drift estimation for stochastic differential equations
We consider non-parametric Bayesian estimation of the drift coefficient of a
one-dimensional stochastic differential equation from discrete-time
observations on the solution of this equation. Under suitable regularity
conditions that are weaker than those previosly suggested in the literature, we
establish posterior consistency in this context. Furthermore, we show that
posterior consistency extends to the multidimensional setting as well, which,
to the best of our knowledge, is a new result in this setting.Comment: 27 page
- …